Device, method, and program for detecting object

ABSTRACT

A device that detects an object includes a receiver that receives information about the object detected by a sensor, multiple circuits that detect the object by performing different detection processes, and a control circuit that controls the circuits. The control circuit detects whether the detection circuits are in an abnormal state, based on a change in a state of the circuits, when the control circuit detects that a first circuit of the circuits is in an abnormal state, the control circuit causes the first circuit to stop a detection process being performed by the first circuit and causes one or more circuits, other than the first circuit, to detect the object by causing the one or more circuits to stop performing detection processes performed by the one or more circuits, and to perform detection processes different from the detection processes being performed by the one or more circuits.

BACKGROUND

1. Technical Field

The present disclosure relates to a device, method, and program thatdetect an object. For example, the present disclosure relates to adevice, method, and program that detect an object from information, suchas an image, in a fail-safe manner.

2. Description of the Related Art

Autonomous cars (robotic cars) driven by machines in place of drivershave been actively studied or commercialized in recent years. One ofelement functions of an autonomous car is a device that detects genericobjects (a generic object detection device). A generic object detectiondevice detects generic objects from information, such as an image,transmitted from a camera device, a distance measurement sensor devicesuch as a radar or stereo camera, or the like. Generic objects includepedestrians, vehicles, road signs, buildings, the areas of roads, andthe like. The generic object detection device of an autonomous car isrequired to detect these objects, and the travel controller of theautonomous car is required to control the body of the autonomous car onthe basis of detection information from the generic object detectiondevice so that the autonomous car safely moves to the destination.

Accordingly, it is essential to design a generic object detection devicein such a manner that it can safely cope with an abnormality that occurstherein. Such a design is called a fail-safe design. Fail-safe is tocontrol a device in the event of an abnormality so that safety is alwaysachieved, and a fail-safe design is one of reliability designs. That is,a fail-safe design assumes that a device or system will inevitably fail.

Japanese Unexamined Patent Application Publication Nos. 2014-21709 and2008-47991 are examples of the related art.

SUMMARY

However, the above disclosed conventional technologies are fail-safetechnologies that cope with a malfunction of a device that acquiresinformation about an object, such as a radar device, and thesetechnologies are not fail-safe technologies relating to a device thatdetects objects (an object detector). For this reason, there has been ademand to further improve fail-safe technologies relating an objectdetector.

In one general aspect, the techniques disclosed here feature a devicefor detecting an object. The device includes a receiver that receivesinformation about the object detected by a sensor; a plurality ofdetection circuits that detect the object from the received informationby performing different detection processes; and a control circuit thatcontrols the detection circuits, wherein the control circuit detectswhether the detection circuits are in an abnormal state, on the basis ofa change in a state of the detection circuits, wherein, when the controlcircuit detects that a first detection circuit of the plurality ofdetection circuits is in an abnormal state, the control circuit causesthe first detection circuit to stop a detection process being performedby the first detection circuit and causes one or more detectioncircuits, other than the first detection circuit, to detect the objectby causing the one or more detection circuits to stop performingdetection processes performed by the one or more detection circuits, andto perform detection processes different from the detection processesbeing performed by the one or more detection circuits.

These general and specific aspects may be implemented using a system, amethod, and a computer program, and any combination of systems, methods,and computer programs.

According to the present disclosure, even if one of the object detectorsis in an abnormal state, the object can be detected in a fail-safemanner with a minimum number of additional elements.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of main components of avehicle having an object detection device of a first embodiment of thepresent disclosure mounted thereon;

FIG. 2 is a block diagram showing an example of the configuration of theobject detection device shown in FIG. 1;

FIG. 3 is a flowchart showing an example of a start-time objectdetection process of the object detection device shown in FIG. 2;

FIG. 4 is a diagram showing an example of a detection process assignmentform used by a detector change unit shown in FIG. 2;

FIG. 5 is a diagram showing an example of an image represented by imageinformation that object detectors shown in FIG. 2 use to perform objectdetection processes;

FIG. 6 is a diagram showing an example of objects detected by the objectdetectors from the image shown in FIG. 5;

FIG. 7 is a diagram showing an example of an instrument image indicatingan autonomous driving state displayed on a display device shown in FIG.1;

FIG. 8 is a flowchart showing an example of an abnormality-time objectdetection process performed when an abnormality occurs in one of theobject detectors of the object detection device shown in FIG. 2;

FIG. 9 is a diagram showing an example of a detection process assignmentform used by the detector change unit when an abnormality occurs in oneof the object detectors shown in FIG. 2;

FIG. 10 is a block diagram showing an example of a fail-safe operationperformed when an abnormality occurs in one of the object detectors ofthe object detection device shown in FIG. 2;

FIG. 11 is a diagram showing an example of an instrument image displayedon the display device when an abnormality occurs in one of the objectdetectors shown in FIG. 2;

FIG. 12 is a diagram showing another example of a detection processassignment form used by the detector change unit when an abnormalityoccurs in one of the object detectors shown in FIG. 2;

FIG. 13 is a block diagram showing an example of the configuration of anobject detection device of a second embodiment of the presentdisclosure;

FIG. 14 is a flowchart showing an example of a start-time objectdetection process of the object detection device shown in FIG. 13;

FIG. 15 is a diagram showing an example of NN storage areas referred toby the object detectors of the object detection device shown in FIG. 13;

FIG. 16 is a flowchart showing an example of an abnormality-time objectdetection process performed when an abnormality occurs in one of theobject detectors of the object detection device shown in FIG. 13; and

FIG. 17 is a diagram showing an example of the NN storage areas referredto by the object detectors of the object detection device shown in FIG.13 when an abnormality occurs in one of the object detectors.

DETAILED DESCRIPTION Underlying Knowledge Forming Basis of the PresentDisclosure

Japanese Unexamined Patent Application Publication No. 2014-21709described above discloses an object position detector that is able toestimate the position of an object in the front-back direction even if aradar device has difficulty in determining the position of the object inthe front-back direction. This object position detector is characterizedin that if a radar failure determination unit determines that the radardevice has failed, it estimates the moving direction of the object usinga camera device and estimates the position of the object in thefront-back direction on the basis of the estimated moving direction. Inother words, this object position detector is a fail-safe technologythat mainly aims to, if a radar sensor device that inputs a beat signalto an object detector fails, switch the radar sensor device to anotherdevice (a camera device) to continue the operation. While JapaneseUnexamined Patent Application Publication No. 2014-21709 discloses thefail-safe technology that copes with a malfunction of a device thatacquires information about objects, such as a radar device, it does notdisclose any fail-safe technology relating to a device that detectsobjects.

Japanese Unexamined Patent Application Publication No. 2008-47991described above discloses an image processing device that even if theenvironment of an area to be monitored changes, is able to improve theobject detection performance using an object detection method suitablefor the environment. This image processing device uses an objectdetector suitable for the environment from among multiple objectdetectors included therein on the basis of information about theenvironment detected by a sensor different from an image sensor. Sincethe image processing device of Japanese Unexamined Patent ApplicationPublication No. 2008-47991 includes the multiple object detectors, itmay be operated in such a manner that an object detector for fail-saferesponds in the event of a failure of one of the object detectors.However, this approach has a problem that a separate object detector forfail-safe is needed. In the field of autonomous cars, there have beenproposed various fail-safe technologies that cope with a failure of acamera device or distance measurement sensor device. However, the objectdetection devices that receive information from the camera device or thelike and detect generic objects may fail as well, and any objectdetector design or technical solution considering fail-safe has not beenconsidered.

In view of the foregoing, the present inventor has conceived of thefollowing improvements.

(1) A device of one aspect of the present disclosure is a device fordetecting an object. The device includes a receiver that receivesinformation about the object detected by a sensor; a plurality ofdetection circuits that detect the object from the received informationby performing different detection processes; and a control circuit thatcontrols the detection circuits, wherein the control circuit detectswhether the detection circuits are in an abnormal state, on the basis ofa change in a state of the detection circuits, wherein, when the controlcircuit detects that a first detection circuit of the plurality ofdetection circuits is in an abnormal state, the control circuit causesthe first detection circuit to stop a detection process being performedby the first detection circuit and causes one or more detectioncircuits, other than the first detection circuit, to detect the objectby causing the one or more detection circuits to stop performingdetection processes performed by the one or more detection circuits, andto perform detection processes different from the detection processesbeing performed by the one or more detection circuits.

According to this configuration, if an abnormality is detected in one ofthe detection circuits, the detection process of the detection circuitin which the abnormality has been detected is stopped, and the detectionprocesses of the normal detection circuits are changed. Thisconfiguration eliminates the need to newly provide a detection circuitfor fail-safe and can cause a normal object detector to perform thedetection process of the detection circuit in which the abnormality hasbeen detected instead. As a result, even if one of the object detectorsis in an abnormal state, the object can be detected in a fail-safemanner with a minimum number of additional elements.

(2) In the above aspect, the information may include one of imageinformation including at least the object and distance informationindicating a distance from the device to the object.

(3) In the above aspect, when the control circuit detects that the firstdetection circuit of the plurality of detection circuits is in anabnormal state, the control circuit causes, on the basis of prioritiesassigned to the different detection processes, a second detectioncircuit to detect the object by causing the second detection circuit toperform a detection process being performed by the first detectioncircuit, the second detection circuit being a detection circuit thatperforms a detection process corresponding to the lowest priority, ofthe one or more detection circuits other than the first detectioncircuit.

According to this configuration, if an abnormality is detected in one ofthe detection circuits, the detection process of a normal detectioncircuit that performs a detection process having the lowest priority, ofthe normal detection circuits is changed to the detection process of thedetection circuit in which the abnormality has been detected. Thus, thenormal detection circuit that performs the detection process having thelowest priority can be caused to perform the detection process of thedetection circuit in which the abnormality has been detected instead,and the detection process having the high priority can be continuedwhile minimizing the reduction in performance associated with thedetection of the object.

(4) In the above aspect, the detection circuits may include at leastthree or more detection circuits. When the control circuit detects thatthe first detection circuit of the plurality of detection circuits is anabnormal state, the control circuit may cause a third detection circuitother than the first detection circuit to perform a detection processcorresponding to the highest priority of the priorities assigned to thedifferent detection processes and may cause a fourth detection circuitother than the first and third detection circuits to detect the objectby causing the fourth detection circuit to perform a detection processcorresponding to the second highest priority of the priorities assignedto the different detection processes.

According to this configuration, if an abnormality is detected in one ofthe detection circuits, one of the normal detection circuits is causedto perform the detection process having the highest priority, and theremaining detection circuit is caused to perform the detection processhaving the second highest priority. Thus, the detection processes havingthe high priorities can be reliably continued.

(5) The detection circuits may include at least three or more detectioncircuits. When the control circuit detects that the first detectioncircuit of the plurality of detection circuits is an abnormal state, thecontrol circuit may cause a third detection circuit other than the firstdetection circuit to perform a detection process corresponding to thehighest priority of the priorities assigned to the different detectionprocesses and may cause a fourth detection circuit other than the firstand third detection circuits to detect the object by causing the fourthdetection circuit to perform a detection process being performed by thefourth detection circuit and the detection process being performed bythe first detection circuit.

According to this configuration, if an abnormality is detected in one ofthe detection circuits, one of the normal detection circuits is causedto perform the detection process having the highest priority, and theremaining detection circuit is caused to perform a detection processbeing performed by the remaining detection circuit as well as thedetection process of the detection circuit in which the abnormality hasbeen detected. Thus, all the detection processes can be continued.

(6) In the above aspect, when the first detection circuit stops thedetection process being performed by the first detection circuit, thecontrol circuit may perform a restoration process for restoring thefirst detection circuit to a normal state.

According to this configuration, when the detection circuit in which theabnormality has been detected is being stopped, the restoration processfor restoring the detection circuit in which the abnormality has beendetected to a normal state is performed. Thus, the detection circuit inwhich the abnormality has been detected can be restored to a normalstate shortly.

(7) In the above aspect, the device may further include a memory. Thecontrol circuit may generate neural networks used in the detectionprocesses performed by the detection circuits using a data group storedin the memory and assigns the neural networks to the respectivedetection circuits. When the control circuit detects that the firstdetection circuit of the plurality of detection circuits is in anabnormal state, the control circuit may cause the one or more detectioncircuits other than the first detection circuit to detect the object bycausing the one or more detection circuits to stop performing detectionprocesses using neural network assigned to the one or more detectioncircuits, and to perform detection processes using neural networksdifferent from neural networks being used in the detection processesbeing performed by the one or more detection circuits.

According to this configuration, neural networks used by the detectioncircuits are generated using a data group for generating neural networksfor detecting the object; the generated neural networks are assigned tothe detection circuits; and the detection circuits detect the objectusing the assigned neural networks. If an abnormality is detected in oneof the detection circuits in this state, the neural networks used by thenormal detection circuits are changed. As a result, it is possible toeliminate the time taken to generate a new neural network to prevent ablank period time from occurring in the object detection process.

(8) In the above aspect, the memory may store the neural networks inareas thereof. When the control circuit detects that the first detectioncircuit of the plurality of detection circuits is in an abnormal state,the control circuit may cause the one or more detection circuits otherthan the first detection circuit to detect the object by causing the oneor more detection circuits to perform detection processes using thedifferent neural networks by changing areas of the memory correspondingto the neural networks being used.

According to this configuration, if an abnormality is detected in one ofthe detection circuits, the neural networks used by the normal detectioncircuits are changed by changing the individual areas of the memoryreferred to by the normal detection circuits. Thus, it is possible toinstantly change the neural networks used by the normal detectioncircuits and to reliably continue the detection processes that use theneural networks.

(9) In the above aspect, the detection circuits may have the samespecifications.

A according to this configuration, it is possible to cause a normaldetection circuit to easily perform the detection process of thedetection circuit in which the abnormality has been detected instead andto reduce the cost of the device.

(10) In the above aspect, the sensor may be a predetermined camera; theinformation may be image information including the object; and thereceiver may receive the image information from the predeterminedcamera.

According to this configuration, generic objects including pedestrians,vehicles, road signs, buildings, the areas of roads, and the like can bedetected using image information obtained by capturing an image ofobjects using the camera.

(11) In the above aspect, the device may be mounted on a vehicle; thevehicle may include a vehicle controller that controls travel of thevehicle; and the control circuit may output a control signal, relatingto the travel of the vehicle to the vehicle controller on the basis of acombination of the detection results of the object by the plurality ofdetection circuits.

According to this configuration, the device is mounted on the vehicle,and the control signal relating to the travel of the vehicle isoutputted to the vehicle controller on the basis of the integrateddetection result of the object by the detection circuits. Thus, even ifan abnormality occurs in a detection circuit, it is possible to detectgeneric objects including pedestrians, other vehicles, road signs,buildings, the areas of roads, and the like necessary to control thetravel of the vehicle and to realize fail-safe autonomous drivingcontrol of the autonomous car using the integrated detection result.

(12) In the above aspect, the plurality of detection circuits mayfurther include a second detection circuit and a third detectioncircuit. The object may include a first object in a first distance rangefrom the vehicle, a second object in a second distance range from thevehicle, and a third object in a third distance range from the vehicle.The first distance range, the second distance range, and the thirddistance range may be closer to the vehicle in this order. The firstdetection circuit may perform a first detection process of detecting thefirst object. The second detection circuit may perform a seconddetection process of detecting the second object. The third detectioncircuit may perform a third detection process of detecting the thirdobject. When the control circuit detects that the first detectioncircuit of the detection circuits is in an abnormal state, the controlcircuit may cause the first detection circuit to stop the first processand causes the third detection circuit to perform the first process.

According to this configuration, if an abnormality is detected in thefirst detection circuit, the detection process of the first detectioncircuit is stopped, and the detection process of the third detectioncircuit is changed in such a manner that the third detection circuitdetects the first object. Thus, even if an abnormality is detected inthe first detection circuit, it is possible to detect the first objectin the first distance range which is closest to the vehicle, forexample, a pedestrian, another vehicle, a road sign, a building, or anarea of a road, and to continue the autonomous driving of the autonomouscar.

The present disclosure can be implemented not only as the above deviceincluding the characteristic elements, but also as a method includingcharacteristic steps corresponding to the characteristic elements of thedevice, and the like. The present disclosure can also be implemented asa storage medium storing a program for causing a processor to performthe characteristic steps included in this method. Accordingly, thefollowing other aspects can also produce effects similar to those of theabove device.

(13) A method of another aspect of the present disclosure is a methodfor detecting an object. The method includes receiving information aboutthe object detected by a sensor; detecting the object from theinformation by performing different detection processes; detectingwhether the detection processes are in an abnormal state, on the basisof a change in a state of the detection processes; when it is detectedthat a first detection process of the detection processes is in anabnormal state, stopping the first detection process and detecting theobject by performing a different detection process, other than the firstdetection process.

(14) A storage medium of yet another aspect of the present disclosure isa non-transitory storage media storing a program for detecting anobject. The program causes a processor to receive information about theobject detected by a sensor; detect the object from the information byperforming different detection processes; detect whether the detectionprocesses are in an abnormal state, on the basis of a change in a stateof the detection processes; when it is detected that a first detectionprocess of the detection processes is in an abnormal state, stop thefirst detection process; and detect the object by causing the processorto perform a different detection process, other than the first detectionprocess.

Of course, the above program can be distributed through a communicationnetwork such as the Internet. A system may be constructed bydistributing some components and others of a device of an embodiment ofthe present disclosure to multiple computers.

It should be noted that embodiments described below are onlyillustrative of the present disclosure. The numbers, shapes, elements,steps, the order of the steps, and the like described in the embodimentsare only illustrative and are not intended to limit the presentdisclosure. Of the elements of the embodiments, elements which are notset forth in the independent claims, which represent the highestconcepts, are described as optional elements. The descriptions of theembodiments may be combined.

Now, object detection device systems of embodiments of the presentdisclosure will be described with reference to the drawings.

First Embodiment

FIG. 1 is a block diagram showing an example of main elements of avehicle having thereon an object detection device of a first embodimentof the present disclosure. As shown in FIG. 1, a vehicle 1 includes acamera device 11, a distance measurement sensor device 12, a travelcontroller 13, a display device 14, and an object detection device 101.The vehicle 1 is an autonomous car that performs autonomous drivingusing the travel controller 13 that controls the travel state and theobject detection device 101 that detects objects. Note that the vehicle1 is not limited to an autonomous car and may be an ordinary vehicledriven by a driver.

The camera device 11 is a charge-coupled device (CCD) image sensor orthe like. It captures an image of various objects (generic objects) infront of the vehicle 1 and outputs image information about the capturedimage of the objects to the object detection device 101. Theconfiguration of the camera device 11 is not limited to this example.Multiple camera devices may capture images of objects on the rear,right, left, and other sides of the vehicle 1.

The distance measurement sensor device 12 is a radar device or the like.It measures the distances from the object detection device 101 tovarious objects (generic objects) in front of the vehicle 1 and outputsdistance information indicating the measured distances to the objectdetection device 101. The configuration of the distance measurementsensor device 12 is not limited to this example. Multiple distancemeasurement sensor devices may measure the distances from the objectdetection device 101 to various objects on the rear, right, left, andother sides of the vehicle 1. Image information obtained by imagingdistance information indicating the distances from the object detectiondevice 101 to objects using a stereo camera or the like may beoutputted.

Since the vehicle 1 is an autonomous car, the object detection device101 detects objects, such as pedestrians, vehicles, road signs,buildings, and the areas of roads, as generic objects and outputs acontrol signal (a drive control signal) based on the object detectionresult to the travel controller 13.

The travel controller 13 is an engine control unit or the like. Itcontrols the travel state of the vehicle 1 on the basis of informationincluding the object detection result of the object detection device101, as well as automatically controls an accelerator operation, a brakeoperation, a steering operation, and the like.

The display device 14 is a liquid crystal display mounted on theinstrument panel of the vehicle 1, or the like. It displays varioustypes of information outputted from the travel controller 13, such asthe state of autonomous driving and the operating state of the objectdetection device 101.

FIG. 2 is a block diagram showing an example of the configuration of theobject detection device shown in FIG. 1. As shown in FIG. 2, the objectdetection device 101 includes an information acquisition unit 102, anobject detection unit 103, a detector abnormality detector 104, adetector change unit 105, and a drive control signal transmitter 106.The object detection unit 103 includes three object detectors, 113, 123,133.

The information acquisition unit 102 acquires, from the camera device11, image information that is obtained by capturing an image of objectsand that serves as object information about objects. It also acquires,from the distance measurement sensor device 12, distance informationindicating the distances from the object detection device 101 to theobjects. It then outputs the acquired image information and distanceinformation to the object detection unit 103. Note that the objectinformation is not limited to the above example and may be any othertypes of information about the objects to be detected.

The object detection unit 103 manages the operation of the objectdetectors 113, 123, 133. Specifically, it provides the image informationand distance information outputted from the information acquisition unit102 to the object detectors 113, 123, 133 to cause the object detectors113, 123, 133 to perform object detection processes. The objectdetectors 113, 123, 133 perform predetermined object detection processesusing the image information and distance information. The objectdetectors 113, 123, 133 preferably have the same performance and sameconfiguration. Note that the number and configuration of the objectdetectors are not limited to the above example and may be changed. Forexample, two or four or more object detectors may be used, or objectdetectors having different performance may be used. The informationprocessed by the object detectors is not limited to the above imageinformation and the like. The object detectors may detect genericobjects from information indicated by detection signals of any types ofsensor devices that use radio waves, heat, sound, infrared rays, or thelike.

The object detectors 123, 133 output the detection results of the objectdetection processes to the object detector 113. The object detector 113integrates the detection result of itself and the detection results ofthe object detectors 123, 133 and outputs the integrated detectionresult to the drive control signal transmitter 106.

The detector abnormality detector 104 detects whether the objectdetectors 113, 123, 133 are in an abnormal state and reports detectorabnormality information reporting an object detector whose abnormalstate has been detected, to the detector change unit 105 and drivecontrol signal transmitter 106. For example, the detector abnormalitydetector 104 detects whether the object detectors 113, 123, 133 are inan abnormal state by monitoring changes in the temperature or powerconsumption of the object detectors 113, 123, 133 in terms of hardwareor by monitoring the frequencies with which the object detectors 113,123, 133 output detection results, in terms of software. Note that whenthe object detectors 113, 123, 133 are operating normally, the detectorabnormality detector 104 may transmit detector normality informationindicating that all the object detectors 113, 123, 133 are normal, tothe drive control signal transmitter 106.

The detector change unit 105 controls the object detection unit 103 tochange the processes of the object detectors 113, 123, 133.Specifically, if detector abnormality detector 104 detects an abnormalstate of one of the object detectors 113, 123, 133, the detector changeunit 105 stops the object detection process of the abnormal objectdetector, whose abnormal state has been detected and changes the objectdetection processes of the normal object detectors, whose abnormalstates have not been detected. When the abnormal object detector isstopping the object detection process, the object detection unit 103performs a restoration process for restoring the abnormal objectdetector to a normal state.

The object detection processes of the object detectors 113, 123, 133have priorities. If the detector abnormality detector 104 detects anabnormal state of one of the object detectors 113, 123, 133, thedetector change unit 105 controls the object detection unit 103 tochange the object detection process of a normal object detector thatperforms an object detection process having the lowest priority, of thenormal object detectors, whose abnormal states have not been detected,to the object detection process of the abnormal object detector, whoseabnormal state has been detected. The detector change unit 105 alsocontrols the object detection unit 103 to cause one of the normal objectdetectors, whose abnormal states have not been detected, to perform anobject detection process having the highest priority and to cause theremaining normal object detector to perform an object detection processhaving the second highest priority.

The objects detected by the object detection device 101 include a firstobject in a first distance range, which is a range closest to thevehicle 1, a second object in a second distance range, which is moredistant from the vehicle 1 than the first distance range, and a thirdobject in a third distance range, which is more distant from the vehicle1 than the second distance range. The object detector 113 detects thefirst object in the first distance range; the object detector 123detects the second object in the second distance range; and the objectdetector 133 detects the third object in the third distance range.

If the detector abnormality detector 104 detects an abnormal state ofthe object detector 113, the detector change unit 105 controls theobject detection unit 103 to stop the object detection process of theobject detector 113 and to change the object detection process of theobject detector 133 so that the object detector 133 detects the firstobject in the first distance range.

Alternatively, if the detector abnormality detector 104 detects anabnormal state of one of the object detectors 113, 123, 133, thedetector change unit 105 may control the object detection unit 103 tocause one of the normal object detectors, whose abnormal states have notbeen detected, to perform an object detection process having the highestpriority and to cause the remaining normal object detector to performthe object detection process being performed by the remaining normalobject detector, as well as the object detection process of the abnormalobject detector, whose abnormal state has been detected.

The drive control signal transmitter 106 outputs a drive control signalto the travel controller 13 on the basis of the integrated detectionresult obtained by integrating the detection results of the objectdetectors 113, 123, 133.

Next, the operation of the object detection device 101 shown in FIG. 2will be described. FIG. 3 is a flowchart showing an example of astart-time object detection process of the object detection device 101shown in FIG. 2. The start-time object detection process shown in FIG. 3is a process performed by the object detection device 101 mounted on thevehicle 1 (autonomous car) when the autonomous driving function isstarted.

First, the detector change unit 105 reports object detection processesto be performed by the object detectors 113, 123, 133 to the objectdetection unit 103, which manages the object detectors 113, 123, 133, onthe basis of a detection process assignment form stored in an internalmemory (not shown) of the detector change unit 105 (step S11).

In this detection process assignment form, detector IDs (identificationnumbers) identifying object detectors, processes, and priorities areassociated with each other in a table format. FIG. 4 is a diagramshowing an example of a detection process assignment form used by thedetector change unit 105 shown in FIG. 2. As shown in FIG. 4, adetection process assignment form T1 consists of detector IDs T2, whichare identifiers associated with the object detectors 113, 123, 133,processes T3 of the detector IDs, and priorities T4 of the processes.

To make the description easy, in the example shown in FIG. 4, “objectdetector 113,” “object detector 123”, and “object detector 133” are usedas the detector IDs T2; “short distance,” “middle distance,” “longdistance,” and “detection result integration” are used as the processesT3; and “1”, “2,” and “3” are used as the priorities T4. As used herein,“short distance” refers to a process of detecting generic objects at aclose distance (in the first distance range) from the vehicle 1.Similarly, “middle distance” refers to a process of detecting genericobjects at a middle distance (in the second distance range) from thevehicle 1, and “long distance” refers to a process of detecting genericobjects at a long distance (in the third distance range) from thevehicle 1. “Detection result integration” refers to a process ofintegrating the detection results of the object detectors 113, 123, 133.The priorities T4 mean that a process having a smaller number has ahigher priority.

For example, on the basis of the detection process assignment form T1shown in FIG. 4, the detector change unit 105 requests the objectdetection unit 103 to set the process of the object detector 113 to“short distance” and “detection result integration” with a priority of“1,” the process of the object detector 123 to “middle distance” with apriority of “2,” and the process of the object detector 133 to “longdistance” with a priority of “3”.

While the detection process assignment form is stored in the detectorchange unit 105 in the above example, it may be stored in a device otherthan the object detection device 101. The object detection unit 103 maystore pair information of the process T3 and priority T4, and thedetector change unit 105 may report pair information of the detector IDT2 and priority T4 to the object detection unit 103. While thepriorities T4 are determined on the basis of the distances from theobject detection device 101 to generic objects, the values of thepriorities may be changed in accordance with the types (person, roadsign, obstacle, animal, and the like) of generic objects to be detectedin the processes T3 (e.g., the priorities may be lowered in the order ofperson, road sign, obstacle, and animal).

The object detectors 113, 123, 133 are object detectors having the sameperformance. As a generic object to be detected is more distant, thegeneric object to be detected is smaller in an image captured by thecamera device 11. For this reason, the amount of calculation of theobject detector 123 is larger than that of the object detector 113, andthe amount of calculation of the object detector 133 is larger than thatof the object detector 123.

Accordingly, in the present embodiment, the object detection processtime per image (per frame) of the object detector 133 is larger thanthose of the object detectors 113, 123. For this reason, in the presentinvention, for example, the information acquisition unit 102 transmitsimages to the object detection unit 103 at 60 frames per second (FPS).As the detection frequencies, the object detector 113 detects images at60 FPS; the object detector 123 at 30 FPS; and the object detector 133at 15 FPS. The object detector 113 integrates the detection results ofthe object detectors 113, 123, 133 at a detection frequency of 60 FPS.

In other words, while the detection result of the object detector 113 isupdated every one frame, the detection result of the object detector 123is updated every two frames and the detection result of the objectdetector 133 is updated every four frames. The distances from the objectdetection device 101 to generic objects are determined on the basis ofdistance information from the distance measurement sensor device 12.Note that if the distance measurement sensor device 12 is not used, thedistances from the object detection device 101 to generic objects may bedetermined on the basis of the sizes of the image areas of the genericobjects in one frame detected by the object detectors 113, 123, 133.

The object detection unit 103 then inputs the image information and thelike from the information acquisition unit 102 to the object detectors113, 123, 133 and causes the object detectors 113, 123, 133 to start toperform object detection processes on the basis of the processes andpriorities described in the report, that is, the detection processassignment form from the detector change unit 105 (step S12). In thiscase, in addition to acquiring the image information from the cameradevice 11, the information acquisition unit 102 may acquire, from thedistance measurement sensor device 12, imaged distance information aboutthe distances from the object detection device 101 to the genericobjects. The object detection unit 103 then transmits the generic objectdetection results from the object detectors 113, 123, 133 to the drivecontrol signal transmitter 106.

FIG. 5 is a diagram showing an example of an image represented by imageinformation that the object detectors 113, 123, 133 shown in FIG. 2 useto perform object detection processes. FIG. 6 is a diagram showing anexample of objects detected by the object detectors 113, 123, 133 fromthe image shown in FIG. 5.

If image information representing a one-frame image P1 shown in FIG. 5is inputted to the information acquisition unit 102 and the objectdetectors 113, 123, 133 perform object detection processes on the imageP1 transmitted from the information acquisition unit 102, imageinformation representing an image equivalent to a detection result imageP2 shown in FIG. 6 is transmitted to the drive control signaltransmitter 106.

In the example shown in FIG. 6, on the basis of the detection processassignment form T1, the object detector 113 detects a passable road areaN1 and another vehicle N2 as short-distance objects; the object detector123 detects a passable road area M1 and a person M2 as middle-distanceobjects; and the object detector 133 detects a passable road area F1 anda road sign F2 as long-distance objects. The object detector 113integrates these detection results and transmits the integrateddetection result to the drive control signal transmitter 106. As seenabove, the object detectors 113, 123, 133, which are assigned theprocesses based on the distances from the object detection device 101,detect generic objects that may be present in the respective distanceranges.

In response to the object detectors 113, 123, 133 starting to performthe object detection processes, the detector abnormality detector 104starts to monitor the object detectors 113, 123, 133 (step S13).Specifically, the detector abnormality detector 104 detects whether theobject detectors 113, 123, 133 are in an abnormal state and reports anobject detector whose abnormal state has been detected, to the detectorchange unit 105 and drive control signal transmitter 106.

The drive control signal transmitter 106 outputs a drive control signalto the travel controller 13 on the basis of the detection result fromthe object detection unit 103, and the travel controller 13 starts toautonomously drive the vehicle 1 on the basis of the drive controlsignal (step S14).

FIG. 7 is a diagram showing an example of an instrument image indicatingthe autonomous driving state displayed on the display device 14 shown inFIG. 1. When the object detectors 113, 123, 133 are operating normally,an instrument image shown in FIG. 7, for example, is displayed on thedisplay device 14 mounted on the instrument panel of the vehicle 1.Since the vehicle 1 is an electric car in the present embodiment, aremaining battery capacity meter BM, a speedometer SM, and a power meterPM, for example, are displayed as an instrument image.

“Cruise Mode” M1 indicating that autonomous driving is being performednormally is displayed as the drive mode under the center of thespeedometer SM; “100% Short” D1 indicating the operating state of theobject detector 113 (a state in which it is performing theshort-distance object detection process at 100% processing capacity) isdisplayed over Ml; “100% Middle” D2 indicating the operating state ofthe object detector 123 (a state in which it is performing themiddle-distance object detection process at 100% processing capacity) isdisplayed over D1; and “100% Long” D3 indicating the operating state ofthe object detector 133 (a state in which it is performing thelong-distance object detection process at 100% processing capacity) isdisplayed over D2.

By displaying an instrument image as described above on the displaydevice 14, the occupant can be notified that autonomous driving is beingperformed and of the states of the object detectors 113, 123, 133.

What is described above is the process performed by the object detectiondevice 101 of the present embodiment at the start of the autonomous car(at the start of the autonomous driving function).

Next, a case in which an abnormality occurs in one of the objectdetectors 113, 123, 133 after starting autonomous driving will bedescribed as an example of the fail-safe operation of the objectdetection device 101. FIG. 8 is a flowchart showing an example of anabnormality-time object detection process performed when an abnormalityoccurs in one of the object detectors 113, 123, 133 of the objectdetection device 101 shown in FIG. 2.

If the detector abnormality detector 104 detects an operationabnormality in one of the object detectors 113, 123, 133, it reportsdetector abnormality information indicating which object detector is inan abnormal state to the detector change unit 105 and drive controlsignal transmitter 106 (step S21). For example, if an abnormality occursin the object detector 113, the detector abnormality detector 104reports detector abnormality information indicating that the objectdetector 113 is in an abnormal state to the detector change unit 105 anddrive control signal transmitter 106.

The detector change unit 105 receives the detector abnormalityinformation, changes the detection process assignment form to be usedfrom the detection process assignment form T1 shown in FIG. 4 to adetection process assignment form in the case where one of the objectdetectors 113, 123, 133 is abnormal, and requests the object detectionunit 103 to change the processes of the object detectors 113, 123, 133so that a fail-safe operation corresponding to this detection processassignment form is performed (step S22). As with the detection processassignment form T1 shown in FIG. 4, the detection process assignmentform in the case where one of the object detectors 113, 123, 133 isabnormal is stored in the internal memory of the detector change unit105 for each of abnormal object detectors. The detector change unit 105reads a detection process assignment form for the abnormal objectdetector from the internal memory and uses it. Note that the detectionprocess assignment forms for abnormal object detectors need not bestored in the internal memory. For example, a detection processassignment form which allows a proper fail-safe operation to beperformed may be generated by the detector change unit 105 or the likeon the basis of the processing capacities of the object detectors 113,123, 133.

FIG. 9 is a diagram showing an example of a detection process assignmentform used by the detector change unit 105 when an abnormality occurs inthe object detector 113 shown in FIG. 2. Processes T3 a and prioritiesT4 a in a detection process assignment form T1 a shown in FIG. 9 differfrom those in the detection process assignment form T1 shown in FIG. 4.Specifically, the process and priority of the object detector 113 arechanged to “restoration process” and “-” (no priority); the process andpriority of the object detector 133 are changed to “short distance,detection result integration” and “1”; and the process and priority ofthe object detector 123 are maintained. As used herein, “restorationprocess” refers to a process of stopping the operation of an abnormalobject detector, whose abnormal state has been detected, and restoringthe abnormal object detector to a normal state.

For example, if an abnormality occurs in the object detector 113, thedetector change unit 105 requests the object detection unit 103 to setthe process of the object detector 113 to “restoration process” with apriority “-,” the process of the object detector 123 to “middledistance” with a priority “2,” and the process of the object detector133 to “short distance” and “detection result integration” with apriority “1,” on the basis of the detection process assignment form T1 ashown in FIG. 9.

The object detection unit 103 changes the processes of the objectdetectors 113, 123, 133 on the basis of the detection process assignmentform in the case where one of the object detectors 113, 123, 133 isabnormal and outputs the integrated detection result of the changedprocesses to the drive control signal transmitter 106 (step S23). Forexample, if an abnormality occurs in the object detector 113, the objectdetection unit 103 performs a “restoration process” for restoring theobject detector 113 to a normal state, on the basis of the process T3 aof the object detector 113. The object detection unit 103 also changesthe object detection process of the object detector 133 to “shortdistance, detection result integration” with “a priority of 1,” on thebasis of the process T3 a and priority T4 a of the object detector 133.

FIG. 10 is a block diagram showing an example of a fail-safe operationperformed when an abnormality occurs in the object detector 113 of theobject detection device 101 shown in FIG. 2. As shown in FIG. 10, if anabnormality occurs in the object detector 113, the object detection unit103 performs a restoration process for restoring the object detector 113to a normal state. The object detector 123 performs a middle-distanceobject detection process with a priority of 2 and outputs the detectionresult to the object detector 133. The object detector 133 performs ashort-distance object detection process with a priority of 1 and adetection result integration process and outputs the integrateddetection result to the drive control signal transmitter 106.

The drive control signal transmitter 106 transmits, to the travelcontroller 13 of the vehicle 1, a drive control signal for performingfail-safe driving, such as the reduction of the travel speed of thevehicle 1 or the stop of the vehicle in a safe place, on the basis ofdetector abnormality information and the integrated detection resultfrom the object detection unit 103 (step S24).

FIG. 11 is a diagram showing an example of an instrument image displayedon the display device 14 when an abnormality occurs in one of the objectdetectors 113, 123, 133 shown in FIG. 2. For example, if an abnormalityoccurs in the object detector 113 and the abnormality-time objectdetection process shown in FIG. 8 is performed, the instrument imageshown in FIG. 11 is displayed on the display device 14 mounted on theinstrument panel of the vehicle 1. In this example, as in FIG. 7, aremaining battery capacity meter BM and a power meter PM are displayedas an instrument image, but the display content of a speedometer SMadiffers from that in FIG. 7.

Specifically, “Fail Safe Mode” M2 indicating that fail-safe driving isbeing performed is displayed as the drive mode under the center of thespeedometer SMa; “0% Recovery” D1 a indicating that the object detector113 is in an abnormal state, an object detection process is stopped, anda restoration process is being performed is displayed over M2; “100%Middle” D2 indicating the operating state of the object detector 123 (astate in which it is performing a middle-distance object detectionprocess at 100% processing capacity) is displayed over D1 a; and “100%Short” D3 a indicating the operating state of the object detector 133 (astate in which it is performing a short-distance object detectionprocess at 100% processing capacity) is displayed over D2.

By displaying an instrument image as described above on the displaydevice 14, the occupant can be notified that the object detector 113 isin the middle of restoration and the vehicle is in a fail-safe drivingstate for avoiding a risk.

The process of the present embodiment described above includes if anabnormal state of one of the object detectors 113, 123, 133 is detected,stopping the object detection process of the abnormal object detector,whose abnormal state has been detected, and changing the objectdetection processes of the normal object detectors, whose abnormalstates have not been detected. Thus, the need to newly provide an objectdetector for fail-safe is eliminated, and the normal object detector canbe caused to perform the object detection process of the abnormal objectdetector instead. As a result, even if one of the object detectors 113,123, 133 is in an abnormal state, objects can be detected in a fail-safemanner with a minimum number of additional elements.

Note that if an abnormal state of one of the object detectors 113, 123,133 is detected, the object detection process of the normal objectdetector, whose abnormal state has not been detected, need not bechanged as shown in FIG. 9 and may be changed in other ways. Forexample, depending on the state or external environment of the vehicle1, the object detection process of the normal object detector may bechanged on the basis of a detection process assignment form as shown inFIG. 12.

FIG. 12 is a diagram showing another example of a detection processassignment form used by the detector change unit 105 when an abnormalityoccurs in the object detector 113 shown in FIG. 2. The processes andpriorities of the object detectors 113, 133 and the priority of theobject detector 123 in a detection process assignment form T1 b shown inFIG. 12 are the same as those in the detection process assignment formT1 a shown in FIG. 9. On the other hand, the process of the objectdetector 123 is changed to a process T3 b, that is, “long distance andshort distance, detector frequency 50%.”

In the present embodiment, a short-distance object detection process isusually performed at a detection frequency of 60 FPS; a middle-distanceobject detection process is usually performed at a detection frequencyof 30 FPS; and a long-distance object detection process is usuallyperformed at a detection frequency of 15 FPS. On the other hand, in theexample shown in FIG. 12, if an abnormality occurs in the objectdetector 113, the object detector 123 detects generic objects on thebasis of the process T3 b thereof in the detection process assignmentform T1 b while allocating half the calculation resources thereof to amiddle-distance object detection process and another half to along-distance object detection process. Thus, the detection frequency ofa short-distance object detection process is changed to 60 FPS; that ofa middle-distance object detection process to 15 FPS; and that of along-distance object detection process to 7 FPS.

As seen above, if an abnormal state of the object detector 113 isdetected, the object detector 133 is caused to perform a short-distanceobject detection process and detection result integration process havingthe highest priority, and the object detector 123 is caused to perform amiddle-distance object detection process being performed thereby, aswell as a long-distance object detection process. Thus, all the objectdetection processes can be continued.

Second Embodiment

FIG. 13 is a block diagram showing an example of the configuration of anobject detection device of a second embodiment of the presentdisclosure. In the present embodiment, an object detection device 101 ausing neural networks (hereafter referred to as “NNs”) in an autonomouscar will be described with reference to FIG. 13. The object detectiondevice 101 a shown in FIG. 13 is basically similar to the objectdetection device 101 shown in FIG. 2 except that it performs objectdetection processes using neural networks described below. For thisreason, the same elements are given the same reference signs and willnot be described repeatedly.

As shown in FIG. 13, the object detection device 101 a includes aninformation acquisition unit 102, an object detection unit 103 a, adetector abnormality detector 104, a detector change unit 105, a drivecontrol signal transmitter 106, and a neural network (NN) storage unit201. The object detection unit 103 a includes three object detectors,113 a, 123 a, 133 a, and a neural network (NN) storage unit 202. The NNstorage unit 202 includes three neural network (NN) storage areas, 213,223, 233, as individual storage areas.

As generic object detectors using NNs, object detectors using NNs thathave performed learning using a technique called “deep learning” havebeen used in recent years. Such object detectors can achieve highdetection performance and have replaced conventional object detectorsthat use manually designed feature values. In the present embodiment,object detection processes are performed using neural networks that haveperformed learning using deep learning as follows.

The NN storage unit 201 stores neural network (NN) generation data as adata group for generating neural networks for detecting objects. As usedherein, “NN generation data” refers to data in which parameters aboutnodes necessary to generate NNs that have performed purpose-specificlearning using the deep learning technique, an inter-node networkconfiguration, and the like are defined. Note that NN generation datamay be acquired in other ways. For example, the NN storage unit 201 maybe omitted, and NN generation data may be acquired from a predeterminedserver or the like through a predetermined network.

The object detection unit 103 a reads the NN generation data from the NNstorage unit 201, generates neural networks to be used by the objectdetectors 113 a, 123 a, 133 a using the NN generation data, and storesthe generated neural networks in the storage areas 213, 223, 233 of theNN storage unit 202. Note that the configuration of the NN storage unit202 is not limited to the above example. For example, the neuralnetworks may be stored in the individual storage areas. Also, the memoryareas (the storage areas 213, 223, 233) storing the generated neuralnetworks may be made redundant to increase fault tolerance and may beduplicated to areas that the object detectors 113 a, 123 a, 133 a canrefer to.

The object detector 103 a manages the operation of the object detectors113 a, 123 a, 133 a and provides image information and distanceinformation outputted from the information acquisition unit 102 to theobject detectors 113 a, 123 a, 133 a. The object detector 103 a alsoassigns the neural networks in the storage areas 213, 223, 233 to theobject detectors 113 a, 123 a, 133 a in accordance with the processes ofthe object detectors 113 a, 123 a, 133 a based on a detection processassignment form from the detector change unit 105.

The object detectors 113 a, 123 a, 133 a perform object detectionprocesses using the image information and distance information from theinformation acquisition unit 102 and the assigned neural networks. Theobject detectors 113 a, 123 a, 133 a are preferably object detectorshaving the same performance and same configuration.

The object detectors 123 a, 133 a output, to the object detector 113 a,the object detection results of the object detection processes using theassigned neural networks. The object detector 113 a integrates thedetection result of itself and the detection results of the objectdetectors 123 a, 133 a and outputs the integrated detection result tothe drive control signal transmitter 106.

The detector abnormality detector 104 detects whether the objectdetectors 113 a, 123 a, 133 a are in an abnormal state and reportsdetector abnormality information reporting an object detector whoseabnormal state has been detected, to the detector change unit 105 anddrive control signal transmitter 106.

The detector change unit 105 controls the object detection unit 103 a tochange the processes of the object detectors 113 a, 123 a, 133 a. Thus,if detector abnormality detector 104 detects an abnormal state of one ofthe object detectors 113 a, 123 a, 133 a, the detector change unit 105stops the object detection process of the abnormal object detector,whose abnormal state has been detected, and changes the object detectionprocesses of the normal object detectors, whose abnormal states have notbeen detected. When the abnormal object detector is stopping the objectdetection process, the object detection unit 103 a performs arestoration process for restoring the abnormal object detector to anormal state.

Also, the detector change unit 105 changes the storage areas 213, 223,233 of the NN storage unit 202 referred to by the normal objectdetectors to change the neural networks used by the normal objectdetectors.

Next, the operation of the object detection device 101 a shown in FIG.13 will be described. FIG. 14 is a flowchart showing an example of astart-time object detection process of the object detection device 101 ashown in FIG. 13. The object detection process shown in FIG. 14 is aprocess performed by the object detection device 101 a mounted on thevehicle 1 (autonomous car) (see FIG. 1) when the autonomous drivingfunction is started.

First, the detector change unit 105 reports object detection processesperformed by the object detectors 113 a, 123 a, 133 a to the objectdetection unit 103 a, which manages the object detectors 113 a, 123 a,133 a, on the basis of a detection process assignment form (e.g., thedetection process assignment form shown in FIG. 4) stored in theinternal memory of the detector change unit 105 (step S31).

Then, the object detector 103 a reads the NN generation data from the NNstorage unit 201 on the basis of the information reported from thedetector change unit 105, generates neural networks for the objectdetectors 113 a, 123 a, 133 a, stores the generated neural networks inthe NN storage areas 213, 223, 233, and assigns the neural networks inthe NN storage areas 213, 223, 233 to the object detectors 113 a, 123 a,133 a (step S32).

FIG. 15 is a diagram showing an example of the NN storage areas 213,223, 233 referred to by the object detectors 113 a, 123 a, 133 a of theobject detection device 101 a shown in FIG. 13. In the example shown inFIG. 15, a neural network for a short-distance object detection processis stored in the NN storage area 213 of the NN storage unit 202; aneural network for a middle-distance object detection process is storedin the NN storage area 223; and a neural network for a long-distanceobject detection process is stored in the NN storage area 233.

At this time, the object detector 103 a assigns the NN storage area 213to the object detector 113 a, the NN storage area 223 to the objectdetector 123 a, the NN storage area 233 to the object detector 133 a.Thus, the object detector 113 a uses the neural network for ashort-distance object detection process with reference to the NN storagearea 213; the object detector 123 a uses the neural network for amiddle-distance object detection process with reference to the NNstorage area 223; and the object detector 133 a uses the neural networkfor a long-distance object detection process with reference to the NNstorage area 233.

The object detection unit 103 a then inputs the image information andthe like from the information acquisition unit 102 to the objectdetectors 113 a, 123 a, 133 a and causes the object detectors 113 a, 123a, 133 a to start to perform object detection processes on the basis ofthe processes and priorities described in the report, that is, thedetection process assignment form from the detector change unit 105using the neural networks in the NN storage areas 213, 223, 233 (stepS33).

In response to the object detection unit 103 a starting to perform theobject detection processes, the detector abnormality detector 104 startsto monitor the object detectors 113 a, 123 a, 133 a (step S34).

Then, the drive control signal transmitter 106 outputs a drive controlsignal to the travel controller 13 on the basis of the detection resultfrom the object detection unit 103 a, and the travel controller 13starts to autonomously drive the vehicle 1 on the basis of the drivecontrol signal (step S35).

What is described above is the process performed by the object detectiondevice 101 a of the present embodiment at the start of the autonomouscar (at the start of the autonomous driving function).

Next, a case in which an abnormality occurs in one of the objectdetectors 113 a, 123 a, 133 a after starting autonomous driving will bedescribed as an example of the fail-safe operation of the objectdetection device 101 a. FIG. 16 is a flowchart showing an example of anabnormality-time object detection process performed when an abnormalityoccurs in one of the object detectors 113 a, 123 a, 133 a of the objectdetection device 101 a shown in FIG. 13.

If the detector abnormality detector 104 detects an operationabnormality in one of the object detectors 113 a, 123 a, 133 a, itreports detector abnormality information indicating which objectdetector is in an abnormal state to the detector change unit 105 anddrive control signal transmitter 106 (step S41). For example, if anabnormality occurs in the object detector 113 a, the detectorabnormality detector 104 reports detector abnormality informationindicating that the object detector 113 a is in an abnormal state to thedetector change unit 105 and drive control signal transmitter 106.

The detector change unit 105 receives the detector abnormalityinformation, changes the detection process assignment form to be usedfrom the detection process assignment form T1 shown in FIG. 4 to adetection process assignment form in the case where one of the objectdetectors 113 a, 123 a, 133 a is abnormal (e.g., the detection processassignment form T1 a shown in FIG. 9), and requests the object detectionunit 103 a to change the processes of the object detectors 113 a, 123 a,133 a so that a fail-safe operation corresponding to this detectionprocess assignment form is performed (step S42).

The object detector 103 a changes the assignment of the neural networksin the NN storage areas 213, 223, 233 to the object detectors 113 a, 123a, 133 a on the basis of the detection process assignment form in thecase where one of the object detectors 113 a, 123 a, 133 a is abnormal,to change the NN storage areas 213, 223, 233 referred to by the objectdetectors 113 a, 123 a, 133 a. Thus, the object detector 103 a changesthe processes of the object detectors 113 a, 123 a, 133 a and outputsthe integrated detection result of the changed processes to the drivecontrol signal transmitter 106 (step S43). For example, if anabnormality occurs in the object detector 113 a, the object detectionunit 103 a performs a “restoration process” for restoring the objectdetector 113 a to a normal state on the basis of the process T3 a of theobject detector 113 a and changes the object detection process of theobject detector 133 a to “short distance, detection result integration”with “a priority of 1” on the basis of the process T3 a and priority T4a of the object detector 133 a.

FIG. 17 is a diagram showing an example of the NN storage areas 213,223, 233 referred to by the object detectors 113 a, 123 a, 133 a of theobject detection device 101 a shown in FIG. 13 when an abnormalityoccurs in one of the object detectors 113 a, 123 a, 133 a. The exampleshown in FIG. 17 shows a case in which an abnormality has occurred inthe object detector 113 a. The object detector 103 a stops the objectdetector 113 a and assigns the NN storage area 223 to the objectdetector 123 a and the NN storage area 213 to the object detector 133 a.

Thus, the object detector 133 a uses a neural network for ashort-distance object detection process with reference to the NN storagearea 213, and the object detector 123 a uses a neural network for amiddle-distance object detection process with reference to the NNstorage area 223.

The drive control signal transmitter 106 transmits, to the travelcontroller 13 of the vehicle 1, a drive control signal for performingfail-safe driving, such as the reduction of the travel speed of thevehicle 1 or the stop of the vehicle in a safe place, on the basis ofthe detector abnormality information and the integrated detection resultfrom the object detection unit 103 a (step S44).

In addition to the effects of the first embodiment, the process of thepresent embodiment described above includes if an abnormal state of oneof the object detectors 113 a, 123 a, 133 a is detected, changing the NNstorage areas 213, 223, 233 referred to by the normal object detectors.Thus, the neural networks used by the normal object detectors, whoseabnormal states have not been detected, are changed. As a result, if anabnormal state of one of the object detectors 113 a, 123 a, 133 a isdetected, it is possible to eliminate the time taken to generate a newneural network to prevent a blank period time from occurring in theobject detection process. Thus, even if an abnormal state of one of theobject detectors 113 a, 123 a, 133 a is detected, it is possible toinstantly change the neural networks used by the normal object detectorsand to reliably continue the object detection processes that use theneural networks. As a result, an interruption of autonomous driving canbe prevented.

While, in the above embodiments, the object detection device of thepresent disclosure is applied to the autonomous car, it can also beapplied to other product fields, such as robots and drones (unmannedflying objects).

The elements described in the embodiments may be implemented assoftware. In this case, the software is stored in a non-transitorystorage medium, such as one or more read-only memories (ROMs), anoptical disc, or a hard disk drive, and when the software is executed bya processor, functions specified by the software are performed by theprocessor and peripheral devices.

The elements described in the embodiments may also be implemented aslarge scale integration (LSI), which is a typical integrated circuit.The LSI may be implemented as individual chips, or part or all thereofmay be implemented as one chip. While the integrated circuit here isreferred to as LSI, it may be referred to as IC (semiconductorintegrated circuit), system LSI, super LSI, or ultra LSI depending onthe degree of integration. The method for forming an integrated circuitis not limited to LSI and may be to use a dedicated circuit orgeneral-purpose processor. After manufacturing LSI, a field programmablegate array (FPGA) may be used, or a reconfigurable processor, which canreconfigure the connection or setting of the circuit cells in LSI, maybe used. If an integrated circuit technology which replaces LSI appearsdue to the progress of the semiconductor technology or due to a derivedtechnology, the elements may be integrated using that technology, as amatter of course.

The object detection device of the present disclosure is able tominimize the addition of elements to a fail-safe object detection deviceand to make further improvements and is useful as a device that detectsobjects.

What is claimed is:
 1. A device for detecting an object, comprising: areceiver that receives information about the object detected by asensor; a plurality of detection circuits that detect the object fromthe received information by performing different detection processes;and a control circuit that controls the detection circuits, wherein thecontrol circuit detects whether the detection circuits are in anabnormal state, on the basis of a change in a state of the detectioncircuits, wherein, when the control circuit detects that a firstdetection circuit of the plurality of detection circuits is in anabnormal state, the control circuit causes the first detection circuitto stop a detection process being performed by the first detectioncircuit and causes one or more detection circuits, other than the firstdetection circuit, to detect the object by causing the one or moredetection circuits to stop performing detection processes performed bythe one or more detection circuits, and to perform detection processesdifferent from the detection processes being performed by the one ormore detection circuits.
 2. The device of claim 1, wherein theinformation includes one of image information including at least theobject and distance information indicating a distance from the device tothe object.
 3. The device of claim 1, wherein when the control circuitdetects that the first detection circuit of the plurality of detectioncircuits is in an abnormal state, the control circuit causes, on thebasis of priorities assigned to the different detection processes, asecond detection circuit to detect the object by causing the seconddetection circuit to perform a detection process being performed by thefirst detection circuit, the second detection circuit being a detectioncircuit that performs a detection process corresponding to the lowestpriority, of the one or more detection circuits other than the firstdetection circuit.
 4. The device of claim 3, wherein the detectioncircuits comprise at least three detection circuits, and when thecontrol circuit detects that the first detection circuit of theplurality of detection circuits is an abnormal state, the controlcircuit causes a third detection circuit other than the first detectioncircuit, to perform a detection process corresponding to the highestpriority of the priorities assigned to the different detection processesand causes a fourth detection circuit, other than the first and thirddetection circuits, to detect the object by causing the fourth detectioncircuit to perform a detection process corresponding to the secondhighest priority of the priorities assigned to the different detectionprocesses.
 5. The device of claim 3, wherein the detection circuitscomprise at least three detection circuits, and when the control circuitdetects that the first detection circuit of the plurality of detectioncircuits is an abnormal state, the control circuit causes a thirddetection circuit other than the first detection circuit, to perform adetection process corresponding to the highest priority of thepriorities assigned to the different detection processes and causes afourth detection circuit, other than the first and third detectioncircuits, to detect the object by causing the fourth detection circuitto perform a detection process being performed by the fourth detectioncircuit and the detection process being performed by the first detectioncircuit.
 6. The device of claim 1, wherein, when the first detectioncircuit stops the detection process being performed by the firstdetection circuit, the control circuit performs a restoration processfor restoring the first detection circuit to a normal state.
 7. Thedevice of claim 1, further comprising a memory, wherein the controlcircuit generates neural networks used in the detection processesperformed by the detection circuits using a data group stored in thememory and assigns the neural networks to the respective detectioncircuits, and when the control circuit detects that the first detectioncircuit of the plurality of detection circuits is in an abnormal state,the control circuit causes the one or more detection circuits, otherthan the first detection circuit, to detect the object by causing theone or more detection circuits to stop performing detection processesusing neural network assigned to the one or more detection circuits, andto perform detection processes using neural networks different fromneural networks being used in the detection processes being performed bythe one or more detection circuits.
 8. The device of claim 7, whereinthe memory stores the neural networks in areas thereof, and when thecontrol circuit detects that the first detection circuit of theplurality of detection circuits is in an abnormal state, the controlcircuit causes the one or more detection circuits, other than the firstdetection circuit, to detect the object by causing the one or moredetection circuits to perform detection processes using the differentneural networks by changing areas of the memory corresponding to theneural networks being used.
 9. The device of claim 1, wherein thedetection circuits have identical specifications.
 10. The device ofclaim 1, wherein the sensor is a predetermined camera, the informationis image information including the object, and the receiver receives theimage information from the predetermined camera.
 11. The device of claim1, wherein the device is mounted on a vehicle, the vehicle comprises avehicle controller that controls travel of the vehicle, and the controlcircuit outputs a control signal, relating to the travel of the vehicle,to the vehicle controller on the basis of a combination of the detectionresults of the object by the plurality of detection circuits.
 12. Thedevice of claim 11, wherein The plurality of detection circuits furthercomprise a second detection circuit and a third detection circuit, theobject comprises a first object in a first distance range from thevehicle, a second object in a second distance range from the vehicle,and a third object in a third distance range from the vehicle, the firstdistance range, the second distance range, and the third distance rangeare closer to the vehicle in this order, the first detection circuitperforms a first detection process of detecting the first object, thesecond detection circuit performs a second detection process ofdetecting the second object, the third detection circuit performs athird detection process of detecting the third object, and when thecontrol circuit detects that the first detection circuit of thedetection circuits is in an abnormal state, the control circuit causesthe first detection circuit to stop the first process and causes thethird detection circuit to perform the first process.
 13. A method fordetecting an object, comprising: receiving information about the objectdetected by a sensor; detecting the object from the information byperforming different detection processes; detecting whether thedetection processes are in an abnormal state, on the basis of a changein a state of the detection processes; when it is detected that a firstdetection process of the detection processes is in an abnormal state,stopping the first detection process and detecting the object byperforming a different detection process, other than the first detectionprocess.
 14. A non-transitory storage media storing a program fordetecting an object, the program causing a processor to: receiveinformation about the object detected by a sensor; detect the objectfrom the information by performing different detection processes; detectwhether the detection processes are in an abnormal state, on the basisof a change in a state of the detection processes; when it is detectedthat a first detection process of the detection processes is in anabnormal state, stop the first detection process; and detect the objectby causing the processor to perform a different detection process, otherthan the first detection process.